A Matrix Framework for Conjugate Gradient Methods and Some Variants of CG with Less Synchronization Overhead

We will present a matrix framework for the conjugate gradient methods, which is expressed in terms of whole vector sequences instead of single vectors or initial parts of sequences. Using this framework extremely concise derivations of the conjugate gradient method, the Lanczos algorithm, and methods such as GMRES, QMR, CGS, can be given. This framework is then used to present some equivalent forms of computing the inner products in the conjugate gradient and Lanczos algorithms. Such equivalent formulations can perform all inner product calculations of a single iteration simultaneously, thereby making the method more efficient in a parallel computing context. 1 Matrix Framework In his 1965 book, Householder [4] presented a short derivation of the conjugate gradient method using a matrix framework. By introducing matrices whose columns are the elements of a vector sequence, e.q. X = (x1, . . .), it becomes possible to express statements about sequences as matrix equations. For instance, a Krylov sequence xi+1 = Axi can be written as AX = XJ where J is the unit lower diagonal matrix (δi,j+1). In [3] this matrix framework is used to give derivations of a number of conjugate gradient-like methods, and to derive basic properties of the methods. As an example of the latter, here is a characterization of how Hessenberg matrices arise in iterative methods: Lemma 1 If A is a square matrix, R a vector sequence, and AR = RH, then H is a non-degenerate upper Hessenberg matrix if and only if the vectors ri are linear combinations of a Krylov sequence obtained from applying A to a multiple of r1. The proof follows from the fact that taking linear combinations corresponds in the matrix framework to right multiplication by an upper triangular matrix. ∗ This work was supported in part by the Applied Mathematical Sciences subprogram of the Office of Energy Research, U.S. Department of Energy under contract DE-AC05-84OR21400 with Martin Marietta Energy Systems, Inc. and in part by DARPA under contract number DAAL0391-C-0047 † Mathematical Sciences Section, Oak Ridge National Laboratory, Oak Ridge, TN 37831–6367. ‡ Computer Science Department, University of Tennessee, Knoxville, TN 37996–1301.